Camera calibration: adjusting camera lense edge distortion

by Uki D. Lucas

Motivation

The raw camera data usually has a certain level of distortion caused by lense shape, this is especially pronounced on the edges of the image. The correction is essential in applications like image recognition used in autonomous vehicles, robotics and even in 3D printing.

Solution approach

The common solution is to compare a known shape object e.g. a chessboard with the image taken, then calculate this specific camera's adjustment parameters that then can be applied to every frame taken by the camera. If the camera changes, the parameters have to be recalibrated.

Get list of sample chessboard images by this camera

In [1]:
# read a list of files using a parern
import glob
image_file_names = glob.glob("camera_cal/calibration*.jpg") # e.g. calibration19.jpg
print("found", len(image_file_names), "image file names" )
print("example", image_file_names[0])
found 20 image file names
example camera_cal/calibration1.jpg
In [2]:
#!/usr/bin/python3
import numpy as np
import cv2
import json
%matplotlib inline

Useful helper method

In [3]:
def plot_images(left_image, right_image):
    import matplotlib.pyplot as plt
    plt.figure(figsize=(20,10))
    plot_image = np.concatenate((left_image, right_image), axis=1)
    plt.imshow(plot_image)
    plt.show() 
In [4]:
def get_sample_gray(image_file_name: str):
    image_original = cv2.imread(image_file_name)
    # convert BGR image to gray-scale
    image_gray = cv2.cvtColor(image_original, cv2.COLOR_BGR2GRAY)
    return image_original, image_gray
    

The pattern will look for "inner corners" e.g. black touching the black square

In [5]:
# Chessboard dimentsions
# nx = 9 # horizontal
# ny = 6 # vertical

def find_inside_corners(image_file_names: list, nx: int=9, ny: int=6):
    # Initialise arrays

    # Object Points: 3d point in real world space
    object_point_list = []

    #Image Points: 2d points in image plane.
    image_points_list = []

    # Generate 3D object points
    object_points = np.zeros((nx*ny, 3), np.float32)
    object_points[:,:2] = np.mgrid[0:nx, 0:ny].T.reshape(-1, 2)

    #print("first 5 elements:\n", object_points[0:5])
    # see: http://docs.opencv.org/trunk/dc/dbb/tutorial_py_calibration.html

    termination_criteria = (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 30, 0.001)
    chessboard_dimentions = (nx, ny)
    import matplotlib.pyplot as plt
    for image_file_name in image_file_names:
        
        print("processing image:", image_file_name)
        image_original, image_gray = get_sample_gray(image_file_name)

        # Find the chess board corners
        # Paramters:
        # - image_gray
        # - the chessboard to be used is 9x6
        # - flags = None
        has_found, corners = cv2.findChessboardCorners(image_gray, chessboard_dimentions, None)
        
        if has_found == True:
            # fill in ObjectPoints
            object_point_list.append(object_points)

            corners2 = cv2.cornerSubPix(image_gray, corners, (11,11), (-1,-1), termination_criteria)
            # fill in ImagePoints
            image_points_list.append(corners2)

            # Draw and display the corners
            # I have to clone/copy the image because cv2.drawChessboardCorners changes the content
            image_corners = cv2.drawChessboardCorners(
                image_original.copy(), 
                chessboard_dimentions, 
                corners2, 
                has_found)

            plot_images(image_original, image_corners)
        else: # not has_found
            print("The", chessboard_dimentions, "chessboard pattern was not found")
            plt.figure()
            plt.imshow(image_original)
            plt.show()
        # end if
    # end for
    return object_point_list, image_points_list
        
In [6]:
object_point_list, image_points_list = find_inside_corners(image_file_names)
processing image: camera_cal/calibration1.jpg
The (9, 6) chessboard pattern was not found
processing image: camera_cal/calibration10.jpg
processing image: camera_cal/calibration11.jpg
processing image: camera_cal/calibration12.jpg
processing image: camera_cal/calibration13.jpg
processing image: camera_cal/calibration14.jpg
processing image: camera_cal/calibration15.jpg
processing image: camera_cal/calibration16.jpg
processing image: camera_cal/calibration17.jpg
processing image: camera_cal/calibration18.jpg
processing image: camera_cal/calibration19.jpg
processing image: camera_cal/calibration2.jpg
processing image: camera_cal/calibration20.jpg
processing image: camera_cal/calibration3.jpg
processing image: camera_cal/calibration4.jpg
The (9, 6) chessboard pattern was not found
processing image: camera_cal/calibration5.jpg
The (9, 6) chessboard pattern was not found
processing image: camera_cal/calibration6.jpg
processing image: camera_cal/calibration7.jpg
processing image: camera_cal/calibration8.jpg
processing image: camera_cal/calibration9.jpg

Calibrate using points

In [7]:
image_original, image_gray = get_sample_gray(image_file_names[1])

# Returns:
# - camera matrix
# - distortion coefficients
# - rotation vectors
# - translation vectors
has_sucess, matrix, distortion, rvecs, tvecs = cv2.calibrateCamera(
    object_point_list, 
    image_points_list, 
    image_gray.shape[::-1], 
    None, 
    None)

This instead of concatinating edges, stretch/curve them

In [8]:
image_dimentions = image_original.shape[:2] # height, width
optimized_matrix, roi = cv2.getOptimalNewCameraMatrix(
    matrix, 
    distortion, 
    image_dimentions, 
    1, 
    image_dimentions)

Remove the distortion form the images

In [9]:
for image_file_name in image_file_names:
    
    print("removing distortion in", image_file_name)
    image_original = cv2.imread(image_file_name)

    print("without edge distortion (cropping)")
    image1 = cv2.undistort(image_original, matrix, distortion, None, None)
    plot_images(image_original, image1)
    
    # save to disk
    cv2.imwrite("output_images/" + "unoptimized_" + image_file_name, image1)
    
    print("with edge distortion (curved)")
    image2 = cv2.undistort(image_original, matrix, distortion, None, optimized_matrix ) 
    plot_images(image_original, image2)
    
    # save to disk
    cv2.imwrite("output_images/" + "optimized_" + image_file_name, image2)
removing distortion in camera_cal/calibration1.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration10.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration11.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration12.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration13.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration14.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration15.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration16.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration17.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration18.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration19.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration2.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration20.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration3.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration4.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration5.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration6.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration7.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration8.jpg
without edge distortion (cropping)
with edge distortion (curved)
removing distortion in camera_cal/calibration9.jpg
without edge distortion (cropping)
with edge distortion (curved)

Conclusion

Assuming that I mask the curved edges, the optimized image provides a better camera calibration because it maintains more area of the image instead of cropping it. This is especially visible in the image camera_cal/calibration2.jpg

In [ ]: